{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xK7B3NnYaPR6"
   },
   "source": [
    "# The TextAttack ecosystem: search, transformations, and constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9rY3w9b2aPSG"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/1_Introduction_and_Transformations.ipynb)\n",
    "\n",
    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/1_Introduction_and_Transformations.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "urhoEHXJf8YK"
   },
   "source": [
    "Installation of  Attack-api branch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HTe13zUKaPSH"
   },
   "source": [
    "An attack in TextAttack consists of four parts.\n",
    "\n",
    "### Goal function\n",
    "\n",
    "The **goal function** determines if the attack is successful or not. One common goal function is **untargeted classification**, where the attack tries to perturb an input to change its classification. \n",
    "\n",
    "### Search method\n",
    "The **search method** explores the space of potential transformations and tries to locate a successful perturbation. Greedy search, beam search, and brute-force search are all examples of search methods.\n",
    "\n",
    "### Transformation\n",
    "A **transformation** takes a text input and transforms it, for example replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.\n",
    "\n",
    "### Constraints\n",
    "Finally, **constraints** determine whether or not a given transformation is valid. Transformations don't perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical  constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tiXXNJO4aPSI"
   },
   "source": [
    "### A custom transformation\n",
    "\n",
    "This lesson explains how to create a custom transformation. In TextAttack, many transformations involve *word swaps*: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create \"typos\" (these don't intend to preserve the grammaticality of inputs). Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.\n",
    "\n",
    "\n",
    "### Banana word swap \n",
    "\n",
    "As an introduction to writing transformations for TextAttack, we're going to try a very simple transformation: one that replaces any given word with the word 'banana'. In TextAttack, there's an abstract `WordSwap` class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend `WordSwap` and implement a single method, `_get_replacement_words`, to indicate to replace each word with 'banana'. 🍌"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "8r7zviXkaPSJ"
   },
   "outputs": [],
   "source": [
    "from textattack.transformations import WordSwap\n",
    "\n",
    "class BananaWordSwap(WordSwap):\n",
    "    \"\"\" Transforms an input by replacing any word with 'banana'.\n",
    "    \"\"\"\n",
    "    \n",
    "    # We don't need a constructor, since our class doesn't require any parameters.\n",
    "\n",
    "    def _get_replacement_words(self, word):\n",
    "        \"\"\" Returns 'banana', no matter what 'word' was originally.\n",
    "        \n",
    "            Returns a list with one item, since `_get_replacement_words` is intended to\n",
    "                return a list of candidate replacement words.\n",
    "        \"\"\"\n",
    "        return ['banana']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RHGvZxenaPSJ"
   },
   "source": [
    "### Using our transformation\n",
    "\n",
    "Now we have the transformation chosen, but we're missing a few other things. To complete the attack, we need to choose the **search method** and **constraints**. And to use the attack, we need a **goal function**, a **model** and a **dataset**. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we'll perform an untargeted attack.)\n",
    "\n",
    "### Creating the goal function, model, and dataset\n",
    "We are performing an untargeted attack on a classification model, so we'll use the `UntargetedClassification` class. For the model, let's use BERT trained for news classification on the AG News dataset. We've pretrained several models and uploaded them to the [HuggingFace Model Hub](https://huggingface.co/textattack). TextAttack integrates with any model from HuggingFace's Model Hub and any dataset from HuggingFace's `datasets`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wREwoDkMaPSK",
    "outputId": "4a8f74c7-c51a-4216-8435-be52d2165d4c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "textattack: Unknown if model of class <class 'transformers.models.bert.modeling_bert.BertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
      "Using custom data configuration default\n",
      "Reusing dataset ag_news (/p/qdata/jy2ma/.cache/textattack/datasets/ag_news/default/0.0.0/0eeeaaa5fb6dffd81458e293dfea1adba2881ffcbdc3fb56baeb5a892566c29a)\n",
      "textattack: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mag_news\u001b[0m, split \u001b[94mtest\u001b[0m.\n"
     ]
    }
   ],
   "source": [
    "# Import the model\n",
    "import transformers\n",
    "from textattack.models.wrappers import HuggingFaceModelWrapper\n",
    "\n",
    "model = transformers.AutoModelForSequenceClassification.from_pretrained(\"textattack/bert-base-uncased-ag-news\")\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"textattack/bert-base-uncased-ag-news\")\n",
    "\n",
    "model_wrapper = HuggingFaceModelWrapper(model, tokenizer)\n",
    "\n",
    "# Create the goal function using the model\n",
    "from textattack.goal_functions import UntargetedClassification\n",
    "goal_function = UntargetedClassification(model_wrapper)\n",
    "\n",
    "# Import the dataset\n",
    "from textattack.datasets import HuggingFaceDataset\n",
    "dataset = HuggingFaceDataset(\"ag_news\", None, \"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sfGMvqcTaPSN"
   },
   "source": [
    "### Creating the attack\n",
    "Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "nSAHSoI_aPSO"
   },
   "outputs": [],
   "source": [
    "from textattack.search_methods import GreedySearch\n",
    "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
    "from textattack import Attack\n",
    "\n",
    "# We're going to use our Banana word swap class as the attack transformation.\n",
    "transformation = BananaWordSwap() \n",
    "# We'll constrain modification of already modified indices and stopwords\n",
    "constraints = [RepeatModification(),\n",
    "               StopwordModification()]\n",
    "# We'll use the Greedy search method\n",
    "search_method = GreedySearch()\n",
    "# Now, let's make the attack from the 4 components:\n",
    "attack = Attack(goal_function, constraints, transformation, search_method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PqrHaZOaaPSO"
   },
   "source": [
    "Let's print our attack to see all the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d2qYOr0maPSP",
    "outputId": "7266dc40-fc6c-4c78-90a8-8150e8fb5d8e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  BananaWordSwap\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(attack)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "m97uyJxDh1wq",
    "outputId": "87ca8836-9781-4c5d-85f2-7ffbf4a7ef80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(OrderedDict([('text', \"Fears for T N pension after talks Unions representing workers at Turner   Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul.\")]), 2)\n"
     ]
    }
   ],
   "source": [
    "print(dataset[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GYKoVFuXaPSP"
   },
   "source": [
    "### Using the attack\n",
    "\n",
    "Let's use our attack to successfully attack 10 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LyokhnFtaPSQ",
    "outputId": "d8a43c4f-1551-40c9-d031-a42b429ed33d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  BananaWordSwap\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ") \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  10%|█         | 1/10 [00:01<00:14,  1.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 1 ---------------------------------------------\n",
      "\u001b[94mBusiness (100%)\u001b[0m --> \u001b[91mWorld (89%)\u001b[0m\n",
      "\n",
      "Fears for T N \u001b[94mpension\u001b[0m after \u001b[94mtalks\u001b[0m \u001b[94mUnions\u001b[0m representing \u001b[94mworkers\u001b[0m at Turner   Newall say they are '\u001b[94mdisappointed'\u001b[0m after talks with stricken parent firm Federal \u001b[94mMogul\u001b[0m.\n",
      "\n",
      "Fears for T N \u001b[91mbanana\u001b[0m after \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m representing \u001b[91mbanana\u001b[0m at Turner   Newall say they are '\u001b[91mbanana\u001b[0m after talks with stricken parent firm Federal \u001b[91mbanana\u001b[0m.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  20%|██        | 2/10 [00:13<00:53,  6.68s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 2 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (64%)\u001b[0m\n",
      "\n",
      "The Race is On: Second Private \u001b[35mTeam\u001b[0m Sets Launch \u001b[35mDate\u001b[0m for \u001b[35mHuman\u001b[0m \u001b[35mSpaceflight\u001b[0m (\u001b[35mSPACE\u001b[0m.\u001b[35mcom\u001b[0m) \u001b[35mSPACE\u001b[0m.\u001b[35mcom\u001b[0m - \u001b[35mTORONTO\u001b[0m, \u001b[35mCanada\u001b[0m -- \u001b[35mA\u001b[0m \u001b[35msecond\u001b[0m\\\u001b[35mteam\u001b[0m of rocketeers \u001b[35mcompeting\u001b[0m for the  #36;10 million Ansari X \u001b[35mPrize\u001b[0m, a \u001b[35mcontest\u001b[0m for\\\u001b[35mprivately\u001b[0m funded \u001b[35msuborbital\u001b[0m \u001b[35mspace\u001b[0m \u001b[35mflight\u001b[0m, has officially \u001b[35mannounced\u001b[0m the first\\\u001b[35mlaunch\u001b[0m date for its \u001b[35mmanned\u001b[0m rocket.\n",
      "\n",
      "The Race is On: Second Private \u001b[91mbanana\u001b[0m Sets Launch \u001b[91mbanana\u001b[0m for \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m (\u001b[91mbanana\u001b[0m.\u001b[91mbanana\u001b[0m) \u001b[91mbanana\u001b[0m.\u001b[91mbanana\u001b[0m - \u001b[91mbanana\u001b[0m, \u001b[91mbanana\u001b[0m -- \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m\\\u001b[91mbanana\u001b[0m of rocketeers \u001b[91mbanana\u001b[0m for the  #36;10 million Ansari X \u001b[91mbanana\u001b[0m, a \u001b[91mbanana\u001b[0m for\\\u001b[91mbanana\u001b[0m funded \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m, has officially \u001b[91mbanana\u001b[0m the first\\\u001b[91mbanana\u001b[0m date for its \u001b[91mbanana\u001b[0m rocket.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 3 / 0 / 0 / 3:  30%|███       | 3/10 [00:18<00:42,  6.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 3 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[94mBusiness (77%)\u001b[0m\n",
      "\n",
      "Ky. Company Wins Grant to \u001b[35mStudy\u001b[0m \u001b[35mPeptides\u001b[0m (\u001b[35mAP\u001b[0m) \u001b[35mAP\u001b[0m - \u001b[35mA\u001b[0m company \u001b[35mfounded\u001b[0m by a \u001b[35mchemistry\u001b[0m \u001b[35mresearcher\u001b[0m at the \u001b[35mUniversity\u001b[0m of Louisville won a grant to develop a method of producing better \u001b[35mpeptides\u001b[0m, which are short chains of \u001b[35mamino\u001b[0m \u001b[35macids\u001b[0m, the building blocks of \u001b[35mproteins\u001b[0m.\n",
      "\n",
      "Ky. Company Wins Grant to \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m (\u001b[94mbanana\u001b[0m) \u001b[94mbanana\u001b[0m - \u001b[94mbanana\u001b[0m company \u001b[94mbanana\u001b[0m by a \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m at the \u001b[94mbanana\u001b[0m of Louisville won a grant to develop a method of producing better \u001b[94mbanana\u001b[0m, which are short chains of \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m, the building blocks of \u001b[94mbanana\u001b[0m.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4:  40%|████      | 4/10 [00:20<00:30,  5.11s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 4 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (65%)\u001b[0m\n",
      "\n",
      "\u001b[35mPrediction\u001b[0m Unit Helps \u001b[35mForecast\u001b[0m Wildfires (AP) \u001b[35mAP\u001b[0m - It's barely dawn when Mike Fitzpatrick \u001b[35mstarts\u001b[0m his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.\n",
      "\n",
      "\u001b[91mbanana\u001b[0m Unit Helps \u001b[91mbanana\u001b[0m Wildfires (AP) \u001b[91mbanana\u001b[0m - It's barely dawn when Mike Fitzpatrick \u001b[91mbanana\u001b[0m his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 5 / 0 / 0 / 5:  50%|█████     | 5/10 [00:22<00:22,  4.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 5 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (62%)\u001b[0m\n",
      "\n",
      "Calif. Aims to Limit Farm-Related \u001b[35mSmog\u001b[0m (AP) AP - Southern California's \u001b[35msmog-fighting\u001b[0m agency went after \u001b[35memissions\u001b[0m of the \u001b[35mbovine\u001b[0m variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.\n",
      "\n",
      "Calif. Aims to Limit Farm-Related \u001b[91mbanana\u001b[0m (AP) AP - Southern California's \u001b[91mbanana\u001b[0m agency went after \u001b[91mbanana\u001b[0m of the \u001b[91mbanana\u001b[0m variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 6 / 0 / 0 / 6:  60%|██████    | 6/10 [00:54<00:36,  9.07s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 6 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (53%)\u001b[0m\n",
      "\n",
      "Open \u001b[35mLetter\u001b[0m Against \u001b[35mBritish\u001b[0m \u001b[35mCopyright\u001b[0m Indoctrination in Schools The \u001b[35mBritish\u001b[0m Department for Education and Skills (DfES) \u001b[35mrecently\u001b[0m \u001b[35mlaunched\u001b[0m a \"\u001b[35mMusic\u001b[0m \u001b[35mManifesto\u001b[0m\" campaign, with the ostensible \u001b[35mintention\u001b[0m of \u001b[35meducating\u001b[0m the \u001b[35mnext\u001b[0m \u001b[35mgeneration\u001b[0m of \u001b[35mBritish\u001b[0m \u001b[35mmusicians\u001b[0m. \u001b[35mUnfortunately\u001b[0m, they also teamed up with the \u001b[35mmusic\u001b[0m industry (\u001b[35mEMI\u001b[0m, and \u001b[35mvarious\u001b[0m \u001b[35martists\u001b[0m) to make this popular. \u001b[35mEMI\u001b[0m has \u001b[35mapparently\u001b[0m \u001b[35mnegotiated\u001b[0m their end well, so that \u001b[35mchildren\u001b[0m in our schools will now be indoctrinated about the illegality of \u001b[35mdownloading\u001b[0m music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.\n",
      "\n",
      "Open \u001b[91mbanana\u001b[0m Against \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m Indoctrination in Schools The \u001b[91mbanana\u001b[0m Department for Education and Skills (DfES) \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m a \"\u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m\" campaign, with the ostensible \u001b[91mbanana\u001b[0m of \u001b[91mbanana\u001b[0m the \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m of \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m. \u001b[91mbanana\u001b[0m, they also teamed up with the \u001b[91mbanana\u001b[0m industry (\u001b[91mbanana\u001b[0m, and \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m) to make this popular. \u001b[91mbanana\u001b[0m has \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m their end well, so that \u001b[91mbanana\u001b[0m in our schools will now be indoctrinated about the illegality of \u001b[91mbanana\u001b[0m music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 6 / 1 / 0 / 7:  70%|███████   | 7/10 [01:47<00:46, 15.36s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 7 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "Loosing the War on Terrorism \\\\\"Sven Jaschan, self-confessed author of the Netsky and Sasser viruses, is\\responsible for 70 percent of virus infections in 2004, according to a six-month\\virus roundup published Wednesday by antivirus company Sophos.\"\\\\\"The 18-year-old Jaschan was taken into custody in Germany in May by police who\\said he had admitted programming both the Netsky and Sasser worms, something\\experts at Microsoft confirmed. (A Microsoft antivirus reward program led to the\\teenager's arrest.) During the five months preceding Jaschan's capture, there\\were at least 25 variants of Netsky and one of the port-scanning network worm\\Sasser.\"\\\\\"Graham Cluley, senior technology consultant at Sophos, said it was staggeri ...\\\\\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 6 / 2 / 0 / 8:  80%|████████  | 8/10 [02:55<00:43, 21.96s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 8 ---------------------------------------------\n",
      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "FOAFKey: FOAF, PGP, Key Distribution, and Bloom Filters \\\\FOAF/LOAF  and bloom filters have a lot of interesting properties for social\\network and whitelist distribution.\\\\I think we can go one level higher though and include GPG/OpenPGP key\\fingerpring distribution in the FOAF file for simple web-of-trust based key\\distribution.\\\\What if we used FOAF and included the PGP key fingerprint(s) for identities?\\This could mean a lot.  You include the PGP key fingerprints within the FOAF\\file of your direct friends and then include a bloom filter of the PGP key\\fingerprints of your entire whitelist (the source FOAF file would of course need\\to be encrypted ).\\\\Your whitelist would be populated from the social network as your client\\discovered new identit ...\\\\\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 7 / 2 / 0 / 9:  90%|█████████ | 9/10 [02:56<00:19, 19.57s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 9 ---------------------------------------------\n",
      "\u001b[35mSci/tech (98%)\u001b[0m --> \u001b[91mWorld (100%)\u001b[0m\n",
      "\n",
      "\u001b[35mE-mail\u001b[0m scam targets police chief Wiltshire Police warns about \"\u001b[35mphishing\u001b[0m\" after its fraud squad chief was targeted.\n",
      "\n",
      "\u001b[91mbanana\u001b[0m scam targets police chief Wiltshire Police warns about \"\u001b[91mbanana\u001b[0m\" after its fraud squad chief was targeted.\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 8 / 2 / 0 / 10: 100%|██████████| 10/10 [02:56<00:00, 17.66s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 10 ---------------------------------------------\n",
      "\u001b[35mSci/tech (98%)\u001b[0m --> \u001b[91mWorld (77%)\u001b[0m\n",
      "\n",
      "Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated \u001b[35mcard\u001b[0m fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.\n",
      "\n",
      "Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated \u001b[91mbanana\u001b[0m fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.\n",
      "\n",
      "\n",
      "\n",
      "+-------------------------------+--------+\n",
      "| Attack Results                |        |\n",
      "+-------------------------------+--------+\n",
      "| Number of successful attacks: | 8      |\n",
      "| Number of failed attacks:     | 2      |\n",
      "| Number of skipped attacks:    | 0      |\n",
      "| Original accuracy:            | 100.0% |\n",
      "| Accuracy under attack:        | 20.0%  |\n",
      "| Attack success rate:          | 80.0%  |\n",
      "| Average perturbed word %:     | 18.71% |\n",
      "| Average num. words per input: | 63.0   |\n",
      "| Avg num queries:              | 934.0  |\n",
      "+-------------------------------+--------+\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
    "from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
    "from textattack.attack_results import SuccessfulAttackResult\n",
    "from textattack import Attacker\n",
    "from textattack import AttackArgs\n",
    "from textattack.datasets import Dataset\n",
    "\n",
    "attack_args = AttackArgs(num_examples=10)\n",
    "\n",
    "attacker = Attacker(attack, dataset, attack_args)\n",
    "\n",
    "attack_results = attacker.attack_dataset()\n",
    "\n",
    "#The following legacy tutorial code shows how the Attack API works in detail.\n",
    "\n",
    "#logger = CSVLogger(color_method='html')\n",
    "\n",
    "#num_successes = 0\n",
    "#i = 0\n",
    "#while num_successes < 10:\n",
    "    #result = next(results_iterable)\n",
    "#    example, ground_truth_output = dataset[i]\n",
    "#    i += 1\n",
    "#    result = attack.attack(example, ground_truth_output)\n",
    "#    if isinstance(result, SuccessfulAttackResult):\n",
    "#        logger.log_attack_result(result)\n",
    "#        num_successes += 1\n",
    "#       print(f'{num_successes} of 10 successes complete.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oRRkNXYmaPSQ"
   },
   "source": [
    "### Visualizing attack results\n",
    "\n",
    "We are logging `AttackResult` objects using a `CSVLogger`. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set `color_method` to `'html'`, the attack results will display their differences, in color, in HTML. Using `IPython` utilities and `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "JafXMELLaPSR",
    "outputId": "48178d1c-5ba9-45f9-b1be-dc6533462c95"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "textattack: Logging to CSV at path results.csv\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original_text</th>\n",
       "      <th>perturbed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fears for T N <font color = blue>pension</font> after <font color = blue>talks</font> <font color = blue>Unions</font> representing <font color = blue>workers</font> at Turner   Newall say they are '<font color = blue>disappointed'</font> after talks with stricken parent firm Federal <font color = blue>Mogul</font>.</td>\n",
       "      <td>Fears for T N <font color = red>banana</font> after <font color = red>banana</font> <font color = red>banana</font> representing <font color = red>banana</font> at Turner   Newall say they are '<font color = red>banana</font> after talks with stricken parent firm Federal <font color = red>banana</font>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Race is On: Second Private <font color = purple>Team</font> Sets Launch <font color = purple>Date</font> for <font color = purple>Human</font> <font color = purple>Spaceflight</font> (<font color = purple>SPACE</font>.<font color = purple>com</font>) <font color = purple>SPACE</font>.<font color = purple>com</font> - <font color = purple>TORONTO</font>, <font color = purple>Canada</font> -- <font color = purple>A</font> <font color = purple>second</font>\\<font color = purple>team</font> of rocketeers <font color = purple>competing</font> for the  #36;10 million Ansari X <font color = purple>Prize</font>, a <font color = purple>contest</font> for\\<font color = purple>privately</font> funded <font color = purple>suborbital</font> <font color = purple>space</font> <font color = purple>flight</font>, has officially <font color = purple>announced</font> the first\\<font color = purple>launch</font> date for its <font color = purple>manned</font> rocket.</td>\n",
       "      <td>The Race is On: Second Private <font color = red>banana</font> Sets Launch <font color = red>banana</font> for <font color = red>banana</font> <font color = red>banana</font> (<font color = red>banana</font>.<font color = red>banana</font>) <font color = red>banana</font>.<font color = red>banana</font> - <font color = red>banana</font>, <font color = red>banana</font> -- <font color = red>banana</font> <font color = red>banana</font>\\<font color = red>banana</font> of rocketeers <font color = red>banana</font> for the  #36;10 million Ansari X <font color = red>banana</font>, a <font color = red>banana</font> for\\<font color = red>banana</font> funded <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font>, has officially <font color = red>banana</font> the first\\<font color = red>banana</font> date for its <font color = red>banana</font> rocket.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ky. Company Wins Grant to <font color = purple>Study</font> <font color = purple>Peptides</font> (<font color = purple>AP</font>) <font color = purple>AP</font> - <font color = purple>A</font> company <font color = purple>founded</font> by a <font color = purple>chemistry</font> <font color = purple>researcher</font> at the <font color = purple>University</font> of Louisville won a grant to develop a method of producing better <font color = purple>peptides</font>, which are short chains of <font color = purple>amino</font> <font color = purple>acids</font>, the building blocks of <font color = purple>proteins</font>.</td>\n",
       "      <td>Ky. Company Wins Grant to <font color = blue>banana</font> <font color = blue>banana</font> (<font color = blue>banana</font>) <font color = blue>banana</font> - <font color = blue>banana</font> company <font color = blue>banana</font> by a <font color = blue>banana</font> <font color = blue>banana</font> at the <font color = blue>banana</font> of Louisville won a grant to develop a method of producing better <font color = blue>banana</font>, which are short chains of <font color = blue>banana</font> <font color = blue>banana</font>, the building blocks of <font color = blue>banana</font>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><font color = purple>Prediction</font> Unit Helps <font color = purple>Forecast</font> Wildfires (AP) <font color = purple>AP</font> - It's barely dawn when Mike Fitzpatrick <font color = purple>starts</font> his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.</td>\n",
       "      <td><font color = red>banana</font> Unit Helps <font color = red>banana</font> Wildfires (AP) <font color = red>banana</font> - It's barely dawn when Mike Fitzpatrick <font color = red>banana</font> his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Calif. Aims to Limit Farm-Related <font color = purple>Smog</font> (AP) AP - Southern California's <font color = purple>smog-fighting</font> agency went after <font color = purple>emissions</font> of the <font color = purple>bovine</font> variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.</td>\n",
       "      <td>Calif. Aims to Limit Farm-Related <font color = red>banana</font> (AP) AP - Southern California's <font color = red>banana</font> agency went after <font color = red>banana</font> of the <font color = red>banana</font> variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Open <font color = purple>Letter</font> Against <font color = purple>British</font> <font color = purple>Copyright</font> Indoctrination in Schools The <font color = purple>British</font> Department for Education and Skills (DfES) <font color = purple>recently</font> <font color = purple>launched</font> a \"<font color = purple>Music</font> <font color = purple>Manifesto</font>\" campaign, with the ostensible <font color = purple>intention</font> of <font color = purple>educating</font> the <font color = purple>next</font> <font color = purple>generation</font> of <font color = purple>British</font> <font color = purple>musicians</font>. <font color = purple>Unfortunately</font>, they also teamed up with the <font color = purple>music</font> industry (<font color = purple>EMI</font>, and <font color = purple>various</font> <font color = purple>artists</font>) to make this popular. <font color = purple>EMI</font> has <font color = purple>apparently</font> <font color = purple>negotiated</font> their end well, so that <font color = purple>children</font> in our schools will now be indoctrinated about the illegality of <font color = purple>downloading</font> music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.</td>\n",
       "      <td>Open <font color = red>banana</font> Against <font color = red>banana</font> <font color = red>banana</font> Indoctrination in Schools The <font color = red>banana</font> Department for Education and Skills (DfES) <font color = red>banana</font> <font color = red>banana</font> a \"<font color = red>banana</font> <font color = red>banana</font>\" campaign, with the ostensible <font color = red>banana</font> of <font color = red>banana</font> the <font color = red>banana</font> <font color = red>banana</font> of <font color = red>banana</font> <font color = red>banana</font>. <font color = red>banana</font>, they also teamed up with the <font color = red>banana</font> industry (<font color = red>banana</font>, and <font color = red>banana</font> <font color = red>banana</font>) to make this popular. <font color = red>banana</font> has <font color = red>banana</font> <font color = red>banana</font> their end well, so that <font color = red>banana</font> in our schools will now be indoctrinated about the illegality of <font color = red>banana</font> music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td><font color = purple>Loosing</font> the <font color = purple>War</font> on <font color = purple>Terrorism</font> \\\\\"<font color = purple>Sven</font> <font color = purple>Jaschan</font>, <font color = purple>self-confessed</font> <font color = purple>author</font> of the <font color = purple>Netsky</font> and <font color = purple>Sasser</font> <font color = purple>viruses</font>, is\\<font color = purple>responsible</font> for <font color = purple>70</font> <font color = purple>percent</font> of <font color = purple>virus</font> <font color = purple>infections</font> in <font color = purple>2004</font>, <font color = purple>according</font> to a <font color = purple>six-month</font>\\<font color = purple>virus</font> <font color = purple>roundup</font> <font color = purple>published</font> <font color = purple>Wednesday</font> by <font color = purple>antivirus</font> <font color = purple>company</font> <font color = purple>Sophos</font>.\"\\\\\"<font color = purple>The</font> <font color = purple>18-year-old</font> <font color = purple>Jaschan</font> was <font color = purple>taken</font> into <font color = purple>custody</font> in <font color = purple>Germany</font> in <font color = purple>May</font> by <font color = purple>police</font> who\\<font color = purple>said</font> he had <font color = purple>admitted</font> <font color = purple>programming</font> both the <font color = purple>Netsky</font> and <font color = purple>Sasser</font> <font color = purple>worms</font>, <font color = purple>something</font>\\<font color = purple>experts</font> at <font color = purple>Microsoft</font> <font color = purple>confirmed</font>. (<font color = purple>A</font> <font color = purple>Microsoft</font> <font color = purple>antivirus</font> <font color = purple>reward</font> <font color = purple>program</font> <font color = purple>led</font> to the\\<font color = purple>teenager's</font> <font color = purple>arrest</font>.) <font color = purple>During</font> the <font color = purple>five</font> <font color = purple>months</font> <font color = purple>preceding</font> <font color = purple>Jaschan's</font> <font color = purple>capture</font>, there\\were at <font color = purple>least</font> <font color = purple>25</font> <font color = purple>variants</font> of <font color = purple>Netsky</font> and <font color = purple>one</font> of the <font color = purple>port-scanning</font> <font color = purple>network</font> <font color = purple>worm</font>\\<font color = purple>Sasser</font>.\"\\\\\"<font color = purple>Graham</font> <font color = purple>Cluley</font>, <font color = purple>senior</font> <font color = purple>technology</font> <font color = purple>consultant</font> at <font color = purple>Sophos</font>, <font color = purple>said</font> it was <font color = purple>staggeri</font> ...\\\\</td>\n",
       "      <td><font color = purple>banana</font> the <font color = purple>banana</font> on <font color = purple>banana</font> \\\\\"<font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font> of the <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>, is\\<font color = purple>banana</font> for <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font> in <font color = purple>banana</font>, <font color = purple>banana</font> to a <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> by <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>.\"\\\\\"<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> was <font color = purple>banana</font> into <font color = purple>banana</font> in <font color = purple>banana</font> in <font color = purple>banana</font> by <font color = purple>banana</font> who\\<font color = purple>banana</font> he had <font color = purple>banana</font> <font color = purple>banana</font> both the <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font>\\<font color = purple>banana</font> at <font color = purple>banana</font> <font color = purple>banana</font>. (<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> to the\\<font color = purple>banana</font> <font color = purple>banana</font>.) <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>, there\\were at <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> and <font color = purple>banana</font> of the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font>.\"\\\\\"<font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> at <font color = purple>banana</font>, <font color = purple>banana</font> it was <font color = purple>banana</font> ...\\\\</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td><font color = purple>FOAFKey</font>: <font color = purple>FOAF</font>, <font color = purple>PGP</font>, <font color = purple>Key</font> <font color = purple>Distribution</font>, and <font color = purple>Bloom</font> <font color = purple>Filters</font> \\\\<font color = purple>FOAF</font>/<font color = purple>LOAF</font>  and <font color = purple>bloom</font> <font color = purple>filters</font> have a <font color = purple>lot</font> of <font color = purple>interesting</font> <font color = purple>properties</font> for <font color = purple>social</font>\\<font color = purple>network</font> and <font color = purple>whitelist</font> <font color = purple>distribution</font>.\\\\<font color = purple>I</font> <font color = purple>think</font> we can <font color = purple>go</font> <font color = purple>one</font> <font color = purple>level</font> <font color = purple>higher</font> <font color = purple>though</font> and <font color = purple>include</font> <font color = purple>GPG</font>/<font color = purple>OpenPGP</font> <font color = purple>key</font>\\<font color = purple>fingerpring</font> <font color = purple>distribution</font> in the <font color = purple>FOAF</font> <font color = purple>file</font> for <font color = purple>simple</font> <font color = purple>web-of-trust</font> <font color = purple>based</font> <font color = purple>key</font>\\<font color = purple>distribution</font>.\\\\<font color = purple>What</font> if we <font color = purple>used</font> <font color = purple>FOAF</font> and <font color = purple>included</font> the <font color = purple>PGP</font> <font color = purple>key</font> <font color = purple>fingerprint</font>(s) for <font color = purple>identities</font>?\\<font color = purple>This</font> <font color = purple>could</font> <font color = purple>mean</font> a <font color = purple>lot</font>.  <font color = purple>You</font> <font color = purple>include</font> the <font color = purple>PGP</font> <font color = purple>key</font> <font color = purple>fingerprints</font> <font color = purple>within</font> the <font color = purple>FOAF</font>\\<font color = purple>file</font> of your <font color = purple>direct</font> <font color = purple>friends</font> and then <font color = purple>include</font> a <font color = purple>bloom</font> <font color = purple>filter</font> of the <font color = purple>PGP</font> <font color = purple>key</font>\\<font color = purple>fingerprints</font> of your <font color = purple>entire</font> <font color = purple>whitelist</font> (the <font color = purple>source</font> <font color = purple>FOAF</font> <font color = purple>file</font> <font color = purple>would</font> of <font color = purple>course</font> <font color = purple>need</font>\\to be <font color = purple>encrypted</font> ).\\\\<font color = purple>Your</font> <font color = purple>whitelist</font> <font color = purple>would</font> be <font color = purple>populated</font> from the <font color = purple>social</font> <font color = purple>network</font> as your <font color = purple>client</font>\\<font color = purple>discovered</font> <font color = purple>new</font> <font color = purple>identit</font> ...\\\\</td>\n",
       "      <td><font color = purple>banana</font>: <font color = purple>banana</font>, <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font>, and <font color = purple>banana</font> <font color = purple>banana</font> \\\\<font color = purple>banana</font>/<font color = purple>banana</font>  and <font color = purple>banana</font> <font color = purple>banana</font> have a <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font> for <font color = purple>banana</font>\\<font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>.\\\\<font color = purple>banana</font> <font color = purple>banana</font> we can <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>/<font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> in the <font color = purple>banana</font> <font color = purple>banana</font> for <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font>.\\\\<font color = purple>banana</font> if we <font color = purple>banana</font> <font color = purple>banana</font> and <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>(s) for <font color = purple>banana</font>?\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> a <font color = purple>banana</font>.  <font color = purple>banana</font> <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> the <font color = purple>banana</font>\\<font color = purple>banana</font> of your <font color = purple>banana</font> <font color = purple>banana</font> and then <font color = purple>banana</font> a <font color = purple>banana</font> <font color = purple>banana</font> of the <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font> of your <font color = purple>banana</font> <font color = purple>banana</font> (the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font>\\to be <font color = purple>banana</font> ).\\\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> be <font color = purple>banana</font> from the <font color = purple>banana</font> <font color = purple>banana</font> as your <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> ...\\\\</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td><font color = purple>E-mail</font> scam targets police chief Wiltshire Police warns about \"<font color = purple>phishing</font>\" after its fraud squad chief was targeted.</td>\n",
       "      <td><font color = red>banana</font> scam targets police chief Wiltshire Police warns about \"<font color = red>banana</font>\" after its fraud squad chief was targeted.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated <font color = purple>card</font> fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.</td>\n",
       "      <td>Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated <font color = red>banana</font> fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
    "\n",
    "logger = CSVLogger(color_method='html')\n",
    "\n",
    "for result in attack_results:\n",
    "    logger.log_attack_result(result)\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yMMF1Vx1aPSR"
   },
   "source": [
    "### Conclusion\n",
    "We can examine these examples for a good idea of how many words had to be changed to \"banana\" to change the prediction score from the correct class to another class. The examples without perturbed words were originally misclassified, so they were skipped by the attack. Looks like some examples needed only a couple \"banana\"s, while others needed up to 17 \"banana\" substitutions to change the class score. Wow! 🍌"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y4MTwyTpaPSR"
   },
   "source": [
    "### Bonus: Attacking Custom Samples\n",
    "\n",
    "We can also attack custom data samples, like these ones I just made up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "L2Po7C8EaPSS",
    "outputId": "d634f038-79e2-4bef-a11e-686a880ce8a7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  BananaWordSwap\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ") \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  25%|██▌       | 1/4 [00:00<00:00,  7.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 1 ---------------------------------------------\n",
      "\u001b[91m0 (96%)\u001b[0m --> \u001b[35m3 (80%)\u001b[0m\n",
      "\n",
      "Malaria \u001b[91mdeaths\u001b[0m in Africa fall by 5% from last year\n",
      "\n",
      "Malaria \u001b[35mbanana\u001b[0m in Africa fall by 5% from last year\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  50%|█████     | 2/4 [00:00<00:00,  3.79it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 2 ---------------------------------------------\n",
      "\u001b[92m1 (98%)\u001b[0m --> \u001b[35m3 (87%)\u001b[0m\n",
      "\n",
      "\u001b[92mWashington\u001b[0m \u001b[92mNationals\u001b[0m \u001b[92mdefeat\u001b[0m the Houston Astros to win the World Series\n",
      "\n",
      "\u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m the Houston Astros to win the World Series\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4: 100%|██████████| 4/4 [00:00<00:00,  4.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------- Result 3 ---------------------------------------------\n",
      "\u001b[94m2 (99%)\u001b[0m --> \u001b[35m3 (94%)\u001b[0m\n",
      "\n",
      "\u001b[94mExxon\u001b[0m \u001b[94mMobil\u001b[0m \u001b[94mhires\u001b[0m a new \u001b[94mCEO\u001b[0m\n",
      "\n",
      "\u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m a new \u001b[35mbanana\u001b[0m\n",
      "\n",
      "\n",
      "--------------------------------------------- Result 4 ---------------------------------------------\n",
      "\u001b[35m3 (93%)\u001b[0m --> \u001b[94m2 (100%)\u001b[0m\n",
      "\n",
      "\u001b[35mMicrosoft\u001b[0m invests $1 billion in OpenAI\n",
      "\n",
      "\u001b[94mbanana\u001b[0m invests $1 billion in OpenAI\n",
      "\n",
      "\n",
      "\n",
      "+-------------------------------+--------+\n",
      "| Attack Results                |        |\n",
      "+-------------------------------+--------+\n",
      "| Number of successful attacks: | 4      |\n",
      "| Number of failed attacks:     | 0      |\n",
      "| Number of skipped attacks:    | 0      |\n",
      "| Original accuracy:            | 100.0% |\n",
      "| Accuracy under attack:        | 0.0%   |\n",
      "| Attack success rate:          | 100.0% |\n",
      "| Average perturbed word %:     | 30.15% |\n",
      "| Average num. words per input: | 8.25   |\n",
      "| Avg num queries:              | 12.75  |\n",
      "+-------------------------------+--------+"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "textattack: Logging to CSV at path results.csv\n",
      "textattack: CSVLogger exiting without calling flush().\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original_text</th>\n",
       "      <th>perturbed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Malaria <font color = red>deaths</font> in Africa fall by 5% from last year</td>\n",
       "      <td>Malaria <font color = purple>banana</font> in Africa fall by 5% from last year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><font color = green>Washington</font> <font color = green>Nationals</font> <font color = green>defeat</font> the Houston Astros to win the World Series</td>\n",
       "      <td><font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> the Houston Astros to win the World Series</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td><font color = blue>Exxon</font> <font color = blue>Mobil</font> <font color = blue>hires</font> a new <font color = blue>CEO</font></td>\n",
       "      <td><font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> a new <font color = purple>banana</font></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><font color = purple>Microsoft</font> invests $1 billion in OpenAI</td>\n",
       "      <td><font color = blue>banana</font> invests $1 billion in OpenAI</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# For AG News, labels are 0: World, 1: Sports, 2: Business, 3: Sci/Tech\n",
    "\n",
    "custom_dataset = [\n",
    "    ('Malaria deaths in Africa fall by 5% from last year', 0),\n",
    "    ('Washington Nationals defeat the Houston Astros to win the World Series', 1),\n",
    "    ('Exxon Mobil hires a new CEO', 2),\n",
    "    ('Microsoft invests $1 billion in OpenAI', 3),\n",
    "]\n",
    "\n",
    "attack_args = AttackArgs(num_examples=4)\n",
    "\n",
    "dataset = Dataset(custom_dataset)\n",
    "\n",
    "attacker = Attacker(attack, dataset, attack_args)\n",
    "\n",
    "results_iterable = attacker.attack_dataset()\n",
    "\n",
    "logger = CSVLogger(color_method='html')\n",
    "\n",
    "for result in results_iterable:\n",
    "    logger.log_attack_result(result)\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "    \n",
    "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "1_Introduction_and_Transformations.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
